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What  is  in  a  name?  Operations  Research,  Business  Analytics, 
Decision  Analytics,  Business  Intelligence,  Advanced  Analytics,  Data 
Science. . .  to  a  certain  degree,  to  label  is  to  limit  -  if  only  intellectually. 
Parody  ensues  when  one  follows  those  whom  one  is  leading.  Just 
as  it  is  difficult  to  move  forward  while  looking  backwards,  there  is 
a  predictably,  cyclic  gyration  as  the  government  attempts  to  capture 
what  industry  is  doing.  By  the  time  bureaucracies  institutionalize  it, 
industry  has  long  since  moved  on.  The  real  element  of  tragedy  here 
is  not  that  this  is  inefficient,  but  rather  that  it  is  entirely  unnecessary. 
The  semantics  of  distinction  convey  a  difference  which  is  seldom 
meaningful.  One  current  manifestation  of  this  general  phenomenon 
can  be  observed  in  regards  to  the  term  Business  Analytics. 

"The  limits  of  my  language  are  the  limits  of  my  mind.  All  I  know  is  what  I 

have  words  for."  -  Ludwig  Wittgenstein 


Defining  Business  Analytics 


One  could  say,  if  absolutes  were  appropriate,  that  making  better 
decisions  is  an  objective  of  "every"  organization.  But  what  exactly 
does  this  mean?  Theoretically,  better  decisions  will  translate  into 
improved  organizational  performance;  however,  it  is  often  difficult 
to  discern  the  underlying  reasons  for  organizational  performance. 

To  gain  insight  into  past  execution  and  to  inform  future  decision¬ 
making,  organizations  explore  and  investigate  past  performance  by 
harnessing  data  and  analytic  techniques. 

"Organizations  -  large  and  small,  private  and  public,  for-profit  and  not-for- 
profit  -  are  using  analytics  to  unlock  the  value  in  their  data,  model  complex 
systems,  and  make  better  decisions  with  less  risk."  -  The  Institute  for  Opera¬ 
tions  Research  and  the  Management  Sciences  (INFORMS) 

This  scientific  process  of  transforming  data  into  insight  with  ana¬ 
lytics  for  better  decision-making  has  taken  the  form  of  various  defi¬ 
nitions.  Regardless  of  the  specific  definition,  they  all  revolve  around 
the  concept  of  aggressively  leveraging  data  and  analytic  techniques  to 
create  evidence-based  decision  making.  This  concept  of  data-driven 
decision  making  has  clearly  been  shown  to  improve  organizational 
performance1. 

Although  analytics  have  been  used  in  organizations  for  a  variety  of 
reasons  for  quite  some  time;  ranging  from  the  simple  (generating  and 
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Figure  1:  Google  trending  of  daily 
searches  for  various  analytic  disciplines 


The  following  are  examples  of  defi¬ 
nitions  for  various  business  analytic 
disciplines: 

Operations  Research:  the  application 
of  scientific  principles  to  business 
management,  providing  a  quantitative 
basis  for  complex  decisions. 

Decision  Analysis:  a  systematic, 
quantitative  and  visual  approach  to 
addressing  and  evaluating  important 
choices  confronted  by  businesses 
Business  Intelligence:  a  variety  of 
applications  used  to  analyze  an  organi¬ 
zation's  raw  data. 

Advanced  Analytics:  a  broad  category 
of  inquiry  that  can  be  used  to  help 
drive  changes  and  improvements  in 
business  practices. 

Data  Science:  the  study  of  where  infor¬ 
mation  comes  from,  what  it  represents 
and  how  it  can  be  turned  into  a  valu¬ 
able  resource  in  the  creation  of  business 
and  IT  strategies. 


1  see  McAfee  &  Brynjolfsson  (2012), 
Chen,  et  al.  (2012),  Brynjolfsson,  et 
al.  (2011),  and  Trkman,  et  al.  (2010)  to 
name  a  few 
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reporting  metrics  &  scorecard  performance)  to  the  more  advanced 
(mathematical  &  statistical  modeling),  many  decision-makers  still 
underutilize  the  available  information  and  power  of  analytics  because 
it  is  either  not  simple  enough  or  arrives  in  an  inconvenient  form2,3. 
This  highlights  the  fact  that,  historically,  the  link  between  an  orga¬ 
nization's  analytic  activities  and  decision-making  has  been  obscure. 
With  no  defined  process  in  place,  quantitative  approaches  have  been 
insufficiently  integrated  with  the  decision-making  process. 

As  a  result,  Davenport4  states  that  Business  Analytics  (BA)  can 
be  defined  as  the  broad  use  of  data  and  quantitative  analysis  for 
decision-making.  It's  the  clearly  defined  process  that  integrates  an¬ 
alytic  techniques  to  make  better  decisions.  It's  the  concept  of  using 
data  and  applying  sound  analytic  methods  to  empower  decision¬ 
makers  in  improving  organizational  performance.  Multiple  defini¬ 
tions  of  BA  have  been  suggested  and  they  all  clearly  state  that  BA  is 
a  process  that  methodically  integrates  the  use  of  analytics  and  data 
in  the  decision-making  process;  it's  the  link  between  analytics  and 
decisions.5 

Establishing  a  Business  Analytics  Process 

The  goal  of  a  business  analytics  process  is  to  turn  data  into 
information,  information  into  insight,  and  then  use  this  insight  to 
make  better  decisions.  But  what  exactly  does  this  process  entail? 

"When  business  analytic  capabilities  are  integrated  into  business  processes, 

decisions  are  more  repeatable,  scalable,  traceable  and  accurate."  -  Gartner,  Inc 

Many  view  the  process  of  analytics  as  a  black  box  and  are  pri¬ 
marily  concerned  only  with  the  end  product.  They  view  it  as  a 
back-office  activity  being  performed  by  "quants"  using  overly  so¬ 
phisticated  math.  This  common  view  leads  to  a  polarized  process  in 
which  interaction  between  analysts,  domain  experts,  and  decision¬ 
makers  are  sparse.  This  leaves  the  analysts  with  little  understanding 
of  how  best  to  model  the  business  problem  at  hand  and  leaves  the 
domain  experts  and  decision-makers  questioning  if  the  model  re¬ 
ally  addresses  their  problem.  To  establish  a  true  business  analytics 
process  an  organization  needs  to  understand  the  analytics  process 
and  analysts  need  to  understand  the  business  and  decision-making 
process. 

To  understand  better,  we  can  segregate  the  BA  process  into  two 
components:  the  scientific  method  of  applying  analytics  and  the  in¬ 
tegration  of  this  analytic  approach  into  the  decision-making  process. 


2  Bartlet,  R.,  2013.  "A  practitioner's 
guide  to  business  analytics:  Using 
data  analysis  tools  to  improve  your 
organization's  decision  making  and 
strategy."  McGraw-Hill. 

3  Davenport,  T.,  et  al.,  2010.  "Analytics 
at  Work."  Harvard  Business  Review 
Press. 

4  Davenport,  T.,  2010.  "The  new  world 
of  business  analytics."  International 
Institute  for  Analytics. 


5  The  following  are  definitions  of  BA: 

BA  refers  to  the  skills,  technologies, 
practices  for  continuous  iterative 
exploration  and  investigation  of  past 
business  performance  to  gain  insight 
and  drive  business  planning. 

BA  is  the  practice  of  iterative,  methodi¬ 
cal  exploration  of  an  organization's  data 
with  emphasis  on  statistical  analysis 
and  is  used  for  data-driven  decision 
making. 

BA  is  the  use  of  analytics,  data,  and 
systemic  reasoning  to  make  business 
decisions 
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How  well  these  two  components  are  orchestrated  will  determine  the 
level  of  success  an  organization  has  in  establishing  a  BA  process. 

"It  is  seen  as  an  end-to-end  process  beginning  with  identifying  the  business 
problem  to  evaluating  and  drawing  conclusions  about  the  prescribed  solution 
arrived  at  through  the  use  of  analytics."  -  INFORMS 


The  Analytic  Method 

The  analytic  process  is  really  just  about  applying  the  scientific  method 
from  a  quantitative  analysis  perspective  to  help  solve  a  problem.  Ev¬ 
ery  problem  being  addressed  from  an  analytic  perspective,  regardless 
of  size,  complexity  or  sophistication,  should  follow  an  organized 
rhythm  that  embodies  the  seven  basic  steps  that  follow. 

Step  1:  Framing  the  business  problem  could  be  the  most  critical 
part  of  the  process.  This  step  requires  full  involvement  of  all  key 
stakeholders  to  outline  the  business  problem  or  decision  being  ad¬ 
dressed,  identify  the  constraints  involved,  define  the  insights  that 
would  benefit  the  decision-maker(s)  the  most,  and  identify  how  ana¬ 
lytics  and  data  could  play  a  role  (or  if  the  problem  is  even  amenable 
to  an  analytics  solution).  Most  importantly,  this  step  needs  to  gain  a 
stakeholder  agreement  on  the  business  problem  statement  outlining 
the  above  key  points. 

"Sound  strategy  starts  with  having  the  right  goal."  -  Michael  Porter 

Step  2:  Now  that  the  problem  statement  has  been  defined,  this 
needs  to  be  reformulated  into  an  analytics  problem  statement.  This 
process  entails  defining  the  key  outputs  required  that  will  empower 
the  decision-maker,  proposing  a  set  of  drivers  and  relationships 
to  the  outputs,  and  outlining  the  assumptions.  It's  important  that 
drivers  and  outputs  defined  in  this  step  are  based  on  the  problem 
needs  and  not  on  current  data  availability. 

"Business  analytics  starts  with  the  business  problem  and  then  looks  for  the 
data.  We  must  avoid  a  structure  that  shifts  our  business  analytics'  focus 
from  seeking  data  based  upon  business  needs  to  offering  only  solutions  made 
possible  by  the  data  available."  -  Randy  Bartlett  (A  Practitioner's  Guide  to 
Business  Analytics) 
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Figure  2:  The  analytic  Process 


This  step  also  needs  to  define  key  metrics  of  success.  To  truly 
understand  if  the  analytic  method  applied  resulted  in  improved 
performance  an  organization  needs  to  define  first  how  to  measure 
and  assess  the  results. 

Step  3:  The  third  step  in  the  process  is  when  the  hands  get  dirty 
and  the  non-analytically  charged  tend  to  lose  interest. 

“Data!  Data!  Data!  I  can't  make  bricks  without  clay!"  -  Sir  Arthur  Conan 

Doyle 
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This  step  involves  identifying  and  prioritizing  data  needs  based 
on  step  2,  acquiring  the  data,  assessing  the  integrity  of  the  data  by 
cleaning  and  preprocessing  the  data,  and  identifying  initial  trends 
and  relationships.  This  step  is  fundamental  in  understanding  the 
data  and,  although  domain  experts  tend  to  become  less  interested 
as  the  technical  analysis  increases,  it  is  paramount  that  they  are  in¬ 
volved  to  help  put  context  around  this  data.  This  step  should  include 
documentation  of  the  data  extraction  and  early  findings  for  repro¬ 
ducibility  and  also  to  share  with  domain  experts  who  can  provide 
more  context  around  the  findings.  Step  3  can  often  lead  the  team  to 
re-define  the  business  and  analytics  problem  statement. 

Step  4:  Now  that  firm  understanding  of  the  data  underlying  the 
business  and  analytic  problem  is  in  place,  an  analytic  methodol¬ 
ogy  that  both  fits  the  data  and  provides  the  outputs  required  by  the 
decision-maker  are  selected. 

Understanding,  and  even  describing,  all  the  analytic  techniques 
available  is  an  intimidating  task.  However,  analytic  techniques  can 
be  categorized  into  three  broad  buckets.  No  one  type  of  analytic 
category  is  better  than  another  and,  in  fact,  they  are  often  used  as 
compliments  to  one  another  to  provide  a  robust  understanding  of  the 
problem. 

Descriptive  analytics  uses  data  aggregation  and  data  mining  tech¬ 
niques  to  provide  insight  into  the  past  and  answer:  "What  has  hap¬ 
pened?"  This  category  primarily  uses  statistics  and  analytics^  that 
describe  the  past  and  are  often  generated  for  reports  and  dashboards  to 
create  benchmarks  and  provide  performance  metrics. 

Predictive  analytic  techniques7  use  knowledge,  usually  extracted  from 
historical  data,  to  predict  future,  or  otherwise  unknown,  events.  The 
goal  is  to  understand  the  future  and  answer:  "What  could  happen?". 

Prescriptive  analytic  methodologies®  not  only  look  into  the  future  to 
predict  likely  outcomes  but  they  also  attempt  to  shape  the  future  by 
optimizing  the  targeted  business  objective  while  balancing  constraints. 
Prescriptive  analytics  are  used  to  advise  on  possible  outcomes  and 
answer:  "What  should  we  do?". 

Although  domain  experts  and  key  stakeholders  will  likely  not  un¬ 
derstand  the  technical  aspects  behind  the  analytic  techniques,  it's  still 
important  that  they  remain  involved  in  this  step  by  understanding 
the  basic  logic  of  the  possible  analytic  techniques  and  the  outputs 
provided  so  that  they  can  provide  input  into  which  techniques  pro¬ 
vide  the  decision-maker  with  the  optimal  insight  to  assist  in  the 
decision  process. 

Step  5:  Once  the  analytic  techniques  are  choosen,  it's  now  time 
to  develop  the  model(s).  This  step  focuses  on  developing  the  model 


Figure  3:  Analytic  categories 


6  Includes  sums,  central  tendencies, 
variances,  percent  changes,  historical 
trends  &  patterns,  correlations,  etc. 

7  Includes  parametric  methods  such  as 
linear  regression,  hierarchical  regres¬ 
sion,  activity-based  costing,  mathemati¬ 
cal  modeling;  simulation  methods  such 
as  discrete  event  simulation  and  agent- 
based  modeling;  classification  methods 
such  as  logistic  regression  and  decision 
trees;  and  artificial  intelligence  methods 
such  as  artificial  neural  networks  and 
bayesian  networks 

8  Includes  optimization  techniques 
such  as  linear  programming,  goal 
programming,  integer /mixed-integer 
programming,  and  search  algorithms; 
artificial  intelligence  optimization 
techniques  such  as  genetic  algorithms 
and  swarm  algorithms;  and  multi¬ 
criteria  decision  models  such  as  analytic 
hierarchy  process,  analytic  network 
process,  multi-attribute  utility  and 
value  theories,  and  value  analysis. 
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structure;  running,  calibrating,  and  validating  the  model;  and  inte¬ 
grating  the  models  if  more  than  one  is  choosen  and  they  feed  into 
one  another.  A  key  component  often  overlooked  in  this  step  is  to 
document  the  modeling  process  for  reproducibility  purposes. 

Step  6:  After  the  model  has  been  developed  to  an  adequate  level 
of  performance,  the  next  step  is  to  deploy  the  model  to  the  user.  The 
deployment  process  can  take  three  general  approaches: 


1.  Automated:  Business  rules  and  IT  architecture  is  in  place  to  allow 
for  automated  reporting.  This  is  often  the  deployment  process  in 
place  for  descriptive  analytics  that  provide  automated  reporting  of 
sales,  profits,  complaints,  etc.9 

2.  Cyclical:  Models  that  are  often  used  in  a  cyclical  nature,  or  not 
requiring  automated  and  real-time  reporting,  are  often  refined 
for  production  use.  This  usually  includes  creating  a  GUI10  that 
allows  the  user  to  interact  with  the  model  in  a  simple  fashion. 
These  models  are  often  used  for  what-if  scenarios  or  in  a  continous 
manner  by  the  decision-maker  to  analyze  the  identified  issue.11 

3.  One-off:  Many  large  investment  decisions  require  an  analytic 
model  but  are  only  used  once.  This  can  include  re-location  de¬ 
cisions,  new  product  life  cycle  sustainment  and  cost  forecasting. 
Often,  these  models  are  not  deployed,  rather,  its  only  the  results 
that  need  to  be  reported.12 

Regardless  of  the  deployment  approach  used,  this  step  also  in¬ 
cludes  delivering  a  report  that  provides  background  on  the  modeling 
process  along  with  findings  / results  that  provide  the  insights  required 
by  the  decision  maker  as  identified  in  step  one. 

Step  7:  The  final  step  is  providing  life  cycle  management  of  the 
model.  If  the  model  deployment  approach  was  designed  for  an  au¬ 
tomated  or  cyclical  approach  then  the  model  should  be  continuously 
reviewed  for  performance  degradation,  to  quickly  recognize  and  act 
on  new  opportunities  such  as  new  data  or  process  options,  and  to 
determine  when  the  model  has  outlived  its  original  purpose. 

Thinking  of  the  analytic  process  in  terms  of  these  seven  steps  cre¬ 
ates  a  comprehensive  framework  that  enables  more  strategic  thinking 
about  analytics  and  how  an  organization  can  treat  them  as  corporate 
assets.  This  framework  focuses  on  the  business  problem  to  guide  the 
analytic  process  and  keeps  the  domain  experts  and  key  stakeholders 
engaged  during  the  entire  process. 


9  Air  Force  examples:  REMIS  reporting 
of  flying  hours  &  sorties,  CRIS  report¬ 
ing  of  budget  execution,  D200  reporting 
of  inventory  levels 

10  General  User  Interface 


11  Air  Force  examples:  ACS  POM  Pri¬ 
oritization  model.  Analysis  of  Mobility 
Platform  (AMP)  model.  Customer 
Oriented  Leveling  Technique  (COLT) 
model.  Weapon  System  Enterprise 
Review  (WSER)  Dashboard 

12  Air  Force  examples:  AF/IMSC  instal¬ 
lation  selection,  KC-46  cost  estimate  for 
POM  decisions 


The  Decision  Making  Process 

The  second  component  required  for  the  BA  process  is  ensuring  the 
analytic  process  is  integrated  into  the  decision-making  process.  To 
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truly  put  analytics  to  work  in  an  enterprise,  analytics  needs  to  be  an 
integral  part  of  everyday  business  decisions  and  processes13.  So  al¬ 
though  organizations  will  have  a  central  department  that  specializes 
on  analytics,  more  often  organizations  are  starting  to  integrate  ana¬ 
lytic  capabilities  throughout  their  enterprise  to  make  analytics  part  of 
their  day-to-day  processes  in  more  domain  areas.14 

Although  the  Analytic  process  keeps  domain  experts  and  key 
stakeholders  heavily  involved,  to  become  an  analytical  enterprise, 
analytics  cannot  be  relegated  to  a  few  analysts  in  a  central  organi¬ 
zation  such  as  A9.  Rather,  analytical  applications  and  tools  must 
become  pervasive  throughout  the  Air  Force  enterprise.  To  achieve 
this,  Gartner's  business  analytics  framework15  stresses  the  use  of 
cross-functional  teams  throughout  the  enterprise.  This  is  a  common 
suggestion16  that  focuses  on  building  "bilingual"  teams  with  busi¬ 
ness/  domain  expertise  along  with  highly  trained  analytic  capabili¬ 
ties.  These  cross-functional  teams  have  the  expertise  to  develop  the 
overall  strategic  plan  and  priorities  for  fitting  analytics  into  the  deci¬ 
sion  processes  within  their  relevant  business  domain  areas.  They  also 
manage  the  programs  and  analytic  process  that  delivers  the  insights 
required,  along  with  proper  interpretation,  for  business  decisions. 

It  is  with  this  heavy  focus  of  integrated  analytics  throughout  the 
enterprise  that  creates  an  analytically  focused  organization  basing 
decisions  on  data-driven  evidence. 

"If  you  can't  measure  something,  you  can't  understand  it.  If  you  can’t  under¬ 
stand  it,  you  can’t  control  it.  If  you  can’t  control  it,  you  can’t  improve  it.”  - 
H.  James  Harrington 


13  Davenport,  T.,  et  al.,  2010.  "Analytics 
at  Work."  Harvard  Business  Review 
Press. 


14  For  example,  P&G  has  a  statistics  de¬ 
partment  but  many  of  their  statisticians 
are  integrated  into  other  departments 
to  provide  analytic  support  for  decision 
processes  (ie  marketing  department  to 
build  a  tool  to  analyze  increasing  claim 
problems,  R&D  department  to  forecast 
new  product  profitibility  potential, 
logistics  department  to  create  routing 
models,  etc.) 

15  Chandler,  N.,  et  al,  2011.  "Garnter's 
business  analytics  framework."  Gartner, 
Inc. 

16  See  Ayres,  I.  2007.  "Super  Crunchers." 
Bantam;  Provost,  F.  &  Fawcett,  T., 

2013.  "Data  Science  for  Business." 
O'Reilly  Media;  Albright,  S.  &  Winston, 
W.,  2014.  "Business  Analytics:  Data 
Analysis  &  Decision  Making."  Cengage 
Learning;  Davenport,  T.  &  Harris,  }., 
2007.  "Competing  on  Analytics:  The 
New  Science  of  Winning."  Harvard 
Business  Review  Press 


Business  Analytics  in  Industry 


It  is  unlikely  that  many  organizations  have  established  an  ana¬ 
lytics  nirvana  that  provides  the  perfect  example  of  what  Business 
Analytics  should  look  like.  However,  examples  are  plenty  to  illustrate 
successful,  and  unsuccessful,  integration  of  analytics  into  business 
decisions  processes. 


Successes 


Successful  integration  of  the  analytics  process  in  decision-making  has 
countless  examples.  This  is  evident  in  numerous  reports17  that  show 
that  analytically  focused  organizations  outperform  those  that  rely 
less  on  analytics,  conferences  that  focus  on  successful  applications  in 
business18,  and  numerous  advanced  degree  programs  that  focus  on 
Business  Analytics19. 


17  McKinsey's  analysis  of  more  than  250 
engagements  over  five  years  revealed 
that  companies  that  put  data  at  the 
center  of  the  marketing  and  sales 
decisions  improve  their  marketing 
return  on  investment  by  15-20%.  See 
footnote  #1  for  more  examples. 

18  The  "Successful  Applications  of 
Customer  Analytics"  Conference 
sponsored  by  Wharton  School  of  Bus. 

19  Master  of  Science  in  Business  Analyt¬ 
ics  at  USC  Marshall  School  of  Business, 
NYU  Stern  School  of  Business,  Kelley 
School  of  Business,  etc. 


UNDERSTANDING  BUSINESS  ANALYTICS  7 


Just  a  few  examples  of  improved  performance  through  Business 
Analytics  includes20: 

•  Progressive  Insurance  began  to  incorporate  analytic  models  in 
their  underwriting  process  to  predict  the  probability  of  accidents 
by  customers.  They  found  that  consumers'  credit  scores  was  a 
surprisingly  good  predictor  of  the  amount  of  accidents  that  a 
customer  would  have.  This  allowed  decision-makers  to  provide 
more  pricing  options  to  customers,  under-cutting  the  competition, 
and  gaining  market  share  of  the  personal  auto  insurance  market.21 

•  Dell  integrated  driver  analysis  and  competitive  benchmarking  to 
identify  key  factors  that  influence  visitors'  purchasing  behavior. 
Through  Bayesian  modeling,  regression  modeling,  and  time-series 
forecasting,  Dell  created  a  more  holistic  buying  experience  for  its 
consumers  and  increased  conversion  rates  and  order  sizes  leading 
to  more  than  $14oM  in  increased  profit  margin  over  two  years.22 

•  Marriott  International  has  expanded  its  optimization  models  orig¬ 
inally  used  to  optimize  guest  room  prices  to  new  areas  such  as 
conference  facilities  and  catering  allowing  managers  to  maximize 
resource  scheduling  and  pricing  which  has  led  to  an  increase  in 
the  actual  revenues-to-maximum  revenues  ratio  of  83%  to  91%. 23 

Failures 

The  goal  of  an  organization  should  not  be  to  just  create  an  enterprise 
focused  on  analytics  but,  rather,  to  create  an  enterprise  focused  on 
doing  analytics  well.  The  same  process  and  logic  errors  that  cause 
people  to  err  without  analytics  can  creep  into  analytically-based 
decisions.  The  following  are  examples  of  errors  caused  by  not  me¬ 
thodically  applying  the  analytic  process  previously  discussed: 


20  100  more  examples  can  be  found 
at  https://www.informs.org/Sites/ 
Getting- Started -With -Analytics/ 
Analytics- Success- Stories/ 

Case- Studies 


21  Davenport,  T.,  et  al.,  2010.  "Analytics 
at  Work."  Harvard  Business  Review 
Press. 


22  https  :  //www.  informs  .  org/Sites/ 
Getting- Started -With -Analytics/ 
Analytics- Success- Stories/ 

Case- Studies/Dell 


23  Davenport,  T.  (2006).  "Competing  on 
Analytics."  Harvard  Business  Review. 
Retrieved  from  http : //www. instare . 
com/pdf/Competing%20on%20Anatytics 
pdf 


•  A  series  of  faulty  assumptions  in  the  analytical  models  used  in 
the  financial  industry  was  a  significant  contributor  to  the  recent 
financial  crisis  of  2007-2008.  Mortgage  lending  models  and  credit 
default  models  were  based  on  faulty  assumptions  of  continuous 
increases  in  housing  prices,  liquidity  in  credit  markets,  and  over¬ 
rated  securities.  Countless  investigations  identified  multiple  prob¬ 
lems  with  the  analytics  behind  the  financial  industry  at  this  time 
to  include  a  lack  of  understanding  of  the  models  by  top  decision¬ 
makers,  faulty  assumptions  biased  by  profit  seeking,  and  lack  of 
model  validation24 

•  In  2003  NASA's  space  shuttle  Columbia  exploded  upon  re-entry. 
Although  this  accident  was  linked  to  physical  damage  to  the  shut¬ 
tle,  the  Columbia  Accident  Investigation  Board25  identified  "sev¬ 
eral  intellectual  failures  in  engineering  analysis".  First,  they  re- 


24  Lewis,  M.  (2011).  "The  big  short: 

Inside  the  doomsday  machine." 

WW  Norton  &  Company;  Hansell, 

S.  "How  wall  street  quants  lided 
to  their  computers."  New  York 
Times.  Retrieved  from  http: //bits, 
blogs . nytimes . com/2008/09/18/ 

how- wall  -  st reets- quants- lied- to- their- computers/. 

25  Columbia  Accident  Investigation 
Board,  Report,  volume  1  (August  2003), 
p.  191. 
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ported  that  reliance  on  past  success  was  often  used  as  a  substitute 
for  sound  analytical  practices.  Second,  over-optimistic  assessments 
were  made  with  lack  of  reasonable  analytic  evidence.  Finally,  as 
information  gets  passed  up  the  NASA  organization  hierarchy, 
from  people  who  do  analysis  to  mid-level  managers  to  high-level 
leadership,  key  explanations  and  supporting  information  is  filtered 
out  by  using  insufficient  reporting  methods.  Edward  Tufte  further 
explains  this  by  illustrating  the  over  reliance  of  PowerPoint  report¬ 
ing  in  the  NASA  organization  rather  than  properly  documenting 
the  analytic  process  through  technical  reports.26 
•  In  addition  to  the  above  process  errors,  excessive  use  of  spread¬ 
sheet  models  can  pose  significant  risks  from  human  error  as  well. 
Examples  include  Fidelity  omitting  a  minus  sign  on  a  net  capital 
loss  of  $1,313  when  cutting  &  pasting  to  a  separate  spreadsheet 
leading  to  an  inaccurate  end-of-year  distribution  announcement, 
a  cut  &  paste  error  in  a  TransAtla  spreadsheet  that  led  to  an  over 
expenditure  of  $24M  on  hedging  contracts,  and  a  typo  in  a  spread¬ 
sheet  formula  by  the  University  of  Toledo  over-projecting  enroll¬ 
ment  along  with  $2.4M  in  unrealized  tuition  funding.27  These 
errors  help  to  emphasize  the  need  for  proper  analytic  processes 
which  include  significant  technical  review,  validation,  and  a  focus 
on  reproducible  analyses. 

Business  Analytics  in  the  Air  Force 

Business  analytics  is  not  new  to  the  Air  Force,  we've  just 
referred  to  it  by  different  names. . .  primarily  just  one. . .  Operations 
Research.  Operations  Research  was  conceived  in  1938  for  the  sole 
purpose  of  defending  Britain  against  Germany  air  attacks  but  the 
scope  and  activities  have  since  greatly  expanded  and  in  many  cases 
resemble  Business  Analytics. 

"[TJrained  in  the  scientific  approach  and  with  the  abilities  to  observe,  to  reason 
from  observation,  to  practice  with  strict  scientific  integrity,  and  to  relate  cause 
to  effect  -  fell  the  role  of  elucidating  the  facts  of  a  situation  and  of  offering 
advice. "  Harold  Larnder2® 

Historical  Applications 

The  turning  point  of  applying  Operations  Research  to  the  wider 
scope  that  is  military  business  operations  occurred  in  1940.  During 
the  French  and  German  battles,  Britain  provided  offensive  military 
support.  During  this  time,  losses  of  British  fighter  aircraft  assets  were 
running  at  high  rates  and  the  French  leadership  was  asking  for  more 


26  Tufte,  E.  (2003).  The  Cognitive  Style 
of  PowerPoint.  Retrieved  from  http: 
//users . ha . uth.gr/tgd/pt0501/09/ 
Tufte.pdf 


27  more  examples  exist  at 

http : //www . cio . com/a  rticle/ 

2438188/enterprise- software/ 

eight- of  -  the -worst- spreadsheet- blunders . 

html 


28  Larnder's  defined  role  of  the  first 
Operations  Research  analysts.  Larnder, 
H.,  1984.  "The  Origin  of  Operational 
Research."  Operations  Research.  Vol. 
32,  No.  2 
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fighter  squadrons  to  provide  support,  which  Churchill  was  likely  to 
approve.  British  Air  Chief  Marshal,  Sir  Hugh  Dowding,  asked  his 
handful  of  Operations  Research  analysts  for  any  scientific  analysis 
that  could  help  inform  this  decision.  A  study  was  performed  that 
analyzed  the  current  daily  losses  and  replacement  rates  to  show  how 
rapidly  the  fighter  air  power  was  being  diminished;  and  that  any 
increase  in  the  loss  rate  would  quickly  reduce  the  fighter  aircraft 
capability  and  strength  to  an  unacceptable  level.  These  findings  were 
transferred  to  graphical  form  and  presented  to  Churchill  dissuading 
him  from  providing  reinforcements  to  France.29 

Since  this  time,  the  scope  and  practices  of  Operations  Research 
have  greatly  expanded  throughout  the  Air  Force.  Simply  assessing 
historical  applications  within  AFMC  provides  an  appreciation  of  the 
scope  of  business  functions  analyzed  and  it  also  confirms  that  much 
of  what  HQ  AFMC/A9A  does  can  be  considered  Business  Analytics. 

" Our  mission  is  to  conduct  Operations  Research  studies  and  analyses  to 
quantify  and  add  insight  into  resource  issues  of  significant  importance  to  the 
Air  Force  Materiel  Command  and  the  Air  Force. "  -  Curtis  E.  Neumann, 

Chief  of  Management  Sciences  Division30 

Just  a  few  of  the  analyses  resembling  Business  Analytics  includes: 

•  Assessing  whether  demands  for  aircraft  recoverable  spares  are 
either  a  function  of  flying  hours,  sorties,  landings,  or  some  combi¬ 
nation  of  the  three  using  regression  analysis.  This  analysis  helped 
decision-makers  understand  at  what  level  these  operational  vari¬ 
ables  influence  demand  levels,  which  in  turn  helps  to  better  plan 
inventory  needs  based  on  planned  operational  activities.31 

•  Evaluating  the  readiness  implications  of  Asset-based  versus 
Rqmts-based  inventory  approaches  by  applying  a  simulation 
model.  This  analysis  identified  how  supply  policy  impacts  asset 
leveling  and  availability  at  bases  and  provides  decision-makers 
with  an  understanding  of  the  pros  and  cons  of  each  policy  ap¬ 
proach.32 

•  Developing  an  interactive  tool  that  assisted  supply  chain  managers 
in  forecasting  and  establishing  metrics  for  issue  effectiveness, 
stockage  effectiveness,  and  expected  wholesale  backorders.  This 
provided  supply  chain  managers  with  a  tool  that  enabled  them  to 
develop  well-founded  forecasts  for  defensible  metric  targets.33 

•  Developing  and  deploying  spares  optimization  algorithm  models 
at  two  test  bases  resulting  in  a  60%  decrease  in  MICAP  hours.  This 
led  decision-makers  to  approving  the  expansion  and  implementa¬ 
tion  of  this  model  to  13  additional  bases  across  6  MAJCOMs.34 

•  Providing  AFMC  leadership  with  a  civilian  pay  and  budget  exe¬ 
cution  dashboard  to  track  $2B  civilian  budget.  This  tool  helped  to 


29  Larnder,  H.,  1984.  "The  Origin  of 
Operational  Research."  Operations 
Research.  Vol.  32,  No.  2 


30  HQ  AFMC,  Management  Sciences  Di¬ 
vision.  (2001).  Fiscal  Year  2001  Annual 
Report.  Retrieved  from  O:\A9A\03  - 
Reading  File\o3-oi  -  Annual  Reports 


31  HQ  AFMC,  Management  Sciences  Di¬ 
vision.  (1994).  Fiscal  Year  1994  Annual 
Report.  Retrieved  from  O:\A9A\03  - 
Reading  File\o3-oi  -  Annual  Reports 


32  HQ  AFMC,  Management  Sciences  Di¬ 
vision.  (1998).  Fiscal  Year  1998  Annual 
Report.  Retrieved  from  O:\A9A\03  - 
Reading  File\o3-oi  -  Annual  Reports 


33  HQ  AFMC,  Management  Sciences  Di¬ 
vision.  (2001).  Fiscal  Year  2001  Annual 
Report.  Retrieved  from  O:\A9A\03  - 
Reading  File\o3-oi  -  Annual  Reports 


34  HQ  AFMC,  Management  Sciences  Di¬ 
vision.  (2004).  Fiscal  Year  2004  Annual 
Report.  Retrieved  from  O:\A9A\03  - 
Reading  File\o3-oi  -  Annual  Reports 
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focus  leadership  on  possible  problem  areas  and  led  to  decisions 
that  improved  execution  by  6%. 35 
•  Developing,  deploying,  and  applying  a  Weapon  System  Sustain¬ 
ment  risk  model  to  the  $156  WSS  portfolio  of  100+  programs. 

This  objective  risk  assessment  provided  insight  to  decision-makers 
through  multiple  "what-if "  scenarios  in  regards  to  $39oM  in  bud¬ 
get  cuts. 36 

Current  Applications 

Recently,  the  concept  of  Business  Analytics  appears  to  be  catching 
on  with  organizations  throughout  the  AFMC  enterprise.  As  is  often 
the  case,  outside  organizations  are  requesting  analytical  support 
from  HQ  AFMC/ A9A;  however,  what  appears  to  differ  is  that  these 
organizations  know  and  understand  that  they  want  to  develop  their 
own  analytic  capability  to  integrate  into  their  day-to-day  processes 
but  they  don't  know  how.  As  a  result  they  are  asking  for  support 
from  HQ  AFMC  /  A9A  to  outline,  develop  and  organize  the  protype 
of  this  organic  analytic  capability.  Three  examples  help  to  illustrate: 


35  HQ  AFMC,  Studies  and  Analyses  Di¬ 
vision.  (2009).  Fiscal  Year  2009  Annual 
Report.  Retrieved  from  O:\A9A\03  - 
Reading  File\o3-oi  -  Annual  Reports 


36  HQ  AFMC,  Studies  and  Analyses  Di¬ 
vision.  (2013).  Fiscal  Year  2013  Annual 
Report.  Retrieved  from  O:\A9A\03  - 
Reading  File\o3-oi  -  Annual  Reports 


The  Air  Force  Installation  Contracting  Agency  (AFICA) 
manages  installation  support  contract  spend  totalling  $ioB  in  fiscal 
year  2013  across  an  enterprise  of  161  installations.  AFICA  wants 
to  establish  a  Business  Analytics  cell  termed  Business  Intelligence 
Competency  Center  (BICC).  The  principal  purpose  of  this  center  is 
to  support  contract  spend  category  definitions,  category  analyses, 
category  sustainment  and  post-initiative  improvement  activities, 
and  on-going  support  activities.  The  AFICA  BICC  will  be  integrated 
into  the  strategic  sourcing  decision-making  process  and  will  provide 
the  analytic  process  for  installation-level  contract  spend  analysis, 
benchmarking,  metric  development  and  analysis,  and  advanced  cost 
analysis  to  relate  contracted  spend  to  causal  factors  and  compare  to 
industry  standards. 

Currently,  the  HQ  AFMC  / A9A  office  is  working  with  AFICA 
to  develop  prototype  research  designs  and  analyses  that  will  help 
define  the  type  of  analytic  work  that  the  BICC  should  perform  to 
help  inform  AFICA  decision-makers.  This  includes: 


2009  2010  2011  2012  2013 


Figure  4:  Obligations  by  Air  Force 
installation-support  contracting  office 


•  Frequency  Analysis:  Develop  a  text  data  mining  algorithm  that 
analyzes  contract  descriptions  and  identifies  the  most  frequent 
descriptors.  This  allows  AFICA  decision-makers  to  categorize  and 
analyze  their  contract  spend  with  more  fidelity.37 

•  Benchmarking  and  Performance  Metrics:  Develop  prototype 
benchmarking  and  performance  metrics  that  allows  AFICA  decision¬ 
makers  to  compare  contracted  spend  across  Air  Force  installations 


37  ie:  Rather  than  simply  categorize 
contracts  as  "Vehicles",  we  can  catego¬ 
rize  Vehicle  contracts  for  more  specific 
purposes  such  as  "Trailers",  "Mobile 
Generators",  "Trucks",  "ATVs",  etc. 
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and  also  against  industry  standards. 

•  Predictive  Analysis:  Identify  potential  drivers  of  contract  spend 
categories  and  test  for  statistically  significant  relationships.  This 
allows  AFICA  decision-makers  to  understand  what  is  influencing 
contract  spend  and  the  cost  savings  that  could  be  realized  if  these 
drivers  can  be  controlled. 

•  BICC  Process  Definition:  HQ  AFMC/A9A  will  help  to  define 
the  resources  and  competencies  required  to  expand  the  prototype 
analysis  into  a  fully  sustained  operation  along  with  outlining  a 
schedule  of  analysis  and  the  business  processes  involved  with  the 
analytic  cell. 


The  Centralized  Asset  Management  (CAM)  office  manages  a 
$15B  portfolio  of  weapon  system  sustainment  requirements  across 
100+  weapon  systems.  The  requirements  are  developed  by  individual 
program  offices  in  conjunction  with  the  respective  Lead  Commands. 
CAM's  role  is  to  validate  and  fund  the  requirements,  aligned  to  Air 
Force  strategic  priorities  across  12  Service  Core  Functions.  Today's 
CAM  processes  tend  to  have  a  program-centric  focus,  without  the 
detailed  analytic  expertise  and  quantitative  tools  to  optimize  funds 
across  the  total  enterprise.  A  Business  Analytics  cell  would  enable 
CAM  to  efficiently  analyze  a  vast  amount  of  historical  and  future 
data  in  order  to  generate  insights  which  would  inform  decisions- 
makers  at  Air  Staff  on  WSS  support  recommendations. 

HQ  AFMC/A9A  has  already  been  heavily  involved  in  defining 
the  resources  required  to  sustain  and  manage  this  Business  Analytic 
capability  along  with  identifying  the  decision  points  that  can  be  in¬ 
fluenced  by  an  integrated  Business  Analytics  process.  Currently,  the 
HQ  AFMC  / A9A  office  is  working  with  CAM  to  develop  prototype 
research  designs,  analyses,  and  tools  to  help  aid  WSS  support  deci¬ 
sions  and  to  establish  an  integrated  Business  Analytic  capability.  This 
includes: 


/  /  /  /  ’  /  /  '  f  / 


Na»lB®lc«i  Bar 


WSS  Task  Level  Requirement  Explorer 


•  POM:  Develop  standard,  repeatable  process  for  analyzing  WSS 
requirements  in  the  POM  and  out-years,  to  include  specific  areas 
of  notable  growth  by  Program  Group,  weapons  system  type  and 
commodity,  along  with  a  detailed  comparison  of  similar  systems. 
This  analytic  foundation  will  inform  AF  Senior  Leaders  of  any 
foreseeable  problems  or  notable  trends  that  may  impact  the  ability 
of  the  AF  to  continue  to  effectively  sustain  weapon  systems  in  the 
future. 

•  Execution  Plan:  Develop  a  real-time,  responsive  analytical  ca¬ 
pability  to  assess  the  viability  of  the  planned  execution  of  WSS 
dollars  in  the  near  term  execution  plan.  This  will  allow  CAM  to 


Figure  5:  Snapshot  of  HQ 
AFMC/AgA's  interactive  CAM  data 
mining  prototype  tool 
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proactively  respond  to  any  foreseeable  issues/ gaps  in  the  exe¬ 
cution  plan  by  realigning  funds  to  meet  the  most  critical  needs, 
potentially  averting  unmitigated  disasters  in  the  year  of  execution. 

•  Risk  Assessment:  Undertake  a  comprehensive  analysis  of  the  in¬ 
puts,  outputs  and  calibration  of  the  WSS  Risk  Assessment  model, 
to  ensure  that  reported  assessments  accurately  reflect  the  true 
risk  being  assumed  in  the  weapon  system  sustainment  portfolio. 

A  detailed  tracking  of  WSS  risk  from  inception  (POM)  through 
planning  and  execution  will  provide  CAM  leadership  with  the 
information  required  to  inform  AF  senior  leaders  of  current  or 
foreseeable  issues  and  trends  related  to  the  ability  of  the  AF  to 
sustain  weapon  systems  in  the  near  and  long  term. 

•  Fund  Spread:  Use  the  Risk  Assessment  model  to  develop  and  as¬ 
sess  the  allocation  of  initial  funds  across  the  100+  Program  Groups 
within  CAM.  This  will  help  ensure  a  reasonable,  risk-based  fund¬ 
ing  outcome  aligned  to  AF  priorities  across  programs  and  Service 
Core  Functions,  and  will  minimize  the  subsequent  funding  adjust¬ 
ments  that  may  be  required  in  the  year  of  execution  to  mitigate 
foreseeable  problems. 

The  Business  and  Enterprise  Services  Division  (AFLCMC/HIC) 
coordinates  the  strategic  sourcing  of  information  technology  (IT)  ser¬ 
vices  along  with  supporting  IT  governance  policies.  In  May  2014 
the  Secretary  of  Air  Force  directed  SAF  /  AQ  to  conduct  a  bend-the- 
cost-curve  (BTCC)  initiative  on  IT.  Led  by  Maj  Gen  Craig  Olson  and 
Mr.  Robert  Shofner,  market  analysis  revealed  three  primary  recom¬ 
mendations,  one  of  which  stated  that  Business  Analytics  is  the  #1 
way  to  reduce  costs.  As  a  result,  they  are  attempting  to  establish  a 
Business  Intelligence  IT  Center  of  Excellence  (BPTCoE)  to  provide 
this  Business  Analytics  capability  with  a  goal  of  reducing  IT  spend  by 
10%  by  the  fiscal  year  2018  POM. 

Currently  HQ  AFMC/ AqA's  role  is  as  consultants  to  help  AFLCMC/HIC 
establish  this  Business  Analytics  capability.  AFLCMC/HIC  has  iden¬ 
tified  several  areas  of  focus  identified  below.  In  addition,  little  infor¬ 
mation  has  been  revealed  on  how  the  Business  Analytic  process  will 
be  sustained,  managed,  or  integrated  into  the  AFLCMC/HI  busi¬ 
ness  process;  this  is  another  areas  where  HQ  AFMC  /  AqA's  expertise 
could  influence  AFLCMC/HIC. 

•  Data:  Identify  the  various  sources  of  data  available  to  feed  into 
the  analytic  process.  In  addition,  address  any  data  quality  and  in¬ 
tegrity  issues.  Thoroughly  understanding  the  data  available  allows 
decision-makers  to  begin  aligning  currently  measured  information, 
and  its  insights,  to  key  AFLCMC/HIC  decision  points. 
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•  Historical  Trend  Analysis:  Develop  analytic  process  to  under¬ 
stand  historical  demands,  costs,  and  refresh  cycles  for  IT  acqui¬ 
sitions  and  services.  This  allows  decision-makers  to  benchmark 
and  compare  across  the  enterprise  to  identify  potential  waste  and 
efficiencies  which  provides  target  cost  levels  to  bend  the  cost  curve 
towards. 

•  Trade-off  Decisions:  Develop  appropriate  analytic  processes  to 
feed  the  governance  process  for  trade-off  decisions.  Although 
vague  at  this  time,  this  focus  area  could  target  similar  analyses  as 
the  CAM  Business  Analytic  process  in  which  analytic  processes  al¬ 
low  for  responsive  capabilities  in  analyzing  POM  versus  execution 
requirements,  assessing  risks,  and  optimizing  current  year  funding 
profiles  across  the  enterprise. 


Towards  an  Analytical  Enterprise 

So,  is  this  concept  of  Business  Analytics  really  new  to  the  Air  Force 
enterprise. . .  no.  After  all,  we've  been  applying  the  analytic  process 
to  inform  operational  decisions  since  World  War  II,  just  under  a  dif¬ 
ferent  name.  What  is  new  is  the  interest  of  analytics  being  integrated 
into  more  areas  of  the  Air  Force  enterprise.  Rather  than  relying  on 
a  central  department  to  be  the  sole  analytic  resource  for  the  enter¬ 
prise,  individual  Air  Force  domain  areas  see  the  value  of  integrating 
analytic  capabilities  into  their  own  decision  processes. 

What  is  important  is  that,  as  an  enterprise,  we  establish  a  common 
understanding  of  what  this  analytic  process  is  and  what  it  represents. 
That,  as  an  enterprise  we  employ  objective  data  and  analyses  as  the 
primary  guides  to  decision-making  and  that  this  requires  a  scien¬ 
tific  method  that  is  integrated  into  business  processes.  Although  we 
must  acknowledge  that  providing  analytic  facts  does  not  necessar¬ 
ily  lead  to  purely  fact-based  decisions  void  of  intuition,  gut  feeling 
or  hearsay;  it's  undeniable  that  creating  an  analytic  culture  that  em¬ 
beds  the  analytic  process  into  its  decision  processes  creates  a  much 
greater  probability  that  future  decisions  will  be  more  informed  and, 
hopefully,  better. 

As  the  central  organization  that  has  decades  of  experience  in  the 
analytic  process  and  has  seen  the  successful  and  unsuccessful  use  of 
analytics  in  the  decision  process,  HQ  AFMC  /  A9 A  has  the  opportu¬ 
nity  to  play  a  critical  role  in  establishing  this  analytic  culture  across 
the  Air  Force  Materiel  Command  and  the  greater  Air  Force  enter¬ 
prise.  We  have  the  opportunity  to  define  this  process  and  provide 
the  direction  required  for  action.  Ultimately,  we  have  the  opportunity 
to  be  the  compass  and  conscious  for  creating  a  Business  Analytics 
culture  across  the  enterprise. 


